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Preface 


This  document  describes  a plan  to  develop  a Conformance  Testing  Service  to  support 
current  product  data  standardization  efforts.  The  Conformance  Testing  Service  is  an 
integral  part  of  an  overall  strategy  for  the  National  PDES  Testbed  at  the  National 
Institute  of  Standards  and  Technology  (NIST).  The  National  PDES  Testbed  was  initiated 
in  1988  under  the  sponsorship  of  ^e  U.S.  Department  of  Defense  Computer-aided 
Acquisition  and  Logistic  Support  (CALS)  Program.  A major  goal  of  the  National  PDES 
Testbed  is  to  provide  technical  leadership  in  a national  effort  to  implement  a complete 
and  useful  specification  for  the  exchange  of  product  data.  This  specification  must  be 
designed  to  meet  the  needs  of  American  industry  and  the  CALS  program. 

The  National  PDES  Testbed  supports  and  actively  participates  in  the  international  effort 
to  develop  the  Standard  for  the  Exchange  of  Product  Model  Data  (STEP).  The  STEP 
development  effort  is  led  by  the  International  Organization  for  Standardization  (ISO) 
Technical  Committee  184  (TC184)  Sub-Committee  4 (SC4). 

This  plan  describes  one  of  several  technical  project  threads  that  have  been  established 
for  the  National  PDES  Testbed.  Other  threads  address  such  areas  as: 

• development  of  configuration  management  systems  and  services, 

• development  of  testing  systems  to  evaluate  the  proposed  standard, 

• specification  and  testing  of  application  protocols, 

• construction  of  a STEP-based  manufacturing  cell,  and 

• establishment  of  the  Product  Data  Exchange  Network. 

The  level  of  support  provided  for  these  technical  threads  and  others  will  be  determined 
by  sponsor  needs  and  a number  of  different  priorities.  As  such,  the  development  plan 
contained  within  this  document  outlines  a reasonable  schedule  to  accomplish  the 
objectives  of  the  thread.  Changes  in  priorities  and  levels  of  support  may  either 
accelerate  or  delay  the  proposed  schedule.  This  plan  will  be  updated  periodically  to 
reflect  technical  changes  in  the  project,  current  level  of  effort,  and  expected  continued 
support. 


Charles  R.  McLean 
CALS  PDES  Project  Manager 
Factory  Automation  Systems  Division 
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Executive  Summary 


The  Standard  for  the  Exchange  of  Product  Model  Data  (STEP)  is  an  international 
standard  which  is  being  developed  to  support  the  product  data  management 
requirements  of  industry.  Different  developers  of  STEP-based  systems  may  implement 
the  standard  in  different  ways.  These  variations  could  lead  to  incompatible  systems, 
thus  defeating  the  purpose  of  the  standard.  Conformance  testing  offers  a solution  to  this 
problem.  Conformance  tests  can  be  used  to  determine  whether  or  not  a particular 
implementation  of  STEP  does  indeed  comply  with  application  protocols  for  STEP. 

This  technical  thread  will  promote  the  establishment  of  an  institutional  framework  for 
conformance  testing  and  develop  methodologies  for  conformance  testing  services 
operating  within  the  framework.  Tools  developed  for  STEP  validation  activities  will 
provide  a foundation  for  the  development  of  conformance  testing  systems.  Conformance 
testing  systems  may  be  developed  by  NIST  and  other  organizations  for  use  by 
conformance  testing  services.  The  institutional  framework  will  be  implemented  at 
approved  conformance  testing  sites  within  the  Product  Data  Exchange  Network  which 
is  being  established  by  NIST  under  the  auspices  of  the  National  PDES  Testbed. 

Conformance  testing  is  the  testing  of  a candidate  product  for  the  existence  of 
characteristics  that  are  required  by  the  standard.  It  helps:  (1)  assure  the  product  meets 
the  requirements  of  the  standard,  (2)  clarify  the  standard  itself  for  implementation, 
(3)  provide  a feedback  loop  to  the  standards-making  bodies  for  improvements  to  the 
standard,  and  (4)  encourage  commercial  development  by  providing  a baseline  for 
commonality  in  all  products.  The  implementation  of  a conformance  testing  system  and 
an  independent  conformance  testing  service  increases  the  likelihood  that  different  STEP 
implementations  will  be  able  to  interoperate,  i.e.,  work  together. 

In  the  conformance  testing  process,  the  Client  is  the  organization  or  individual  seeking 
assurance  that  the  Client's  product  complies  with  the  standard.  With  the  successful 
completion  of  conformance  testing  on  a Client's  STEP  implementation,  the  Client  may 
obtain  a certificate  of  validation.  This  certificate  may  provide  the  Client  with  a market 
advantage:  (1)  it  may  permit  the  Client  to  bid  on  a government  contract  or  (2)  show  a 
potential  government  user  that  the  Client's  product  has  been  tested  under  a controlled 
environment  by  an  independent  Testing  Laboratory.  This  formal  process  improves  the 
competitive  edge  for  the  Client  over  those  implementors  who  have  not  gone  through  the 
same  process. 

The  Conformance  Testing  Service  thread  of  the  National  PDES  Testbed  will:  (1)  promote 
the  construction  of  conformance  testing  tools  and  systems  by  appropriate  organizations 
and  projects,  (2)  develop  test  procedures  and  data  that  adhere  to  STEP  specifications,  (3) 
specify  the  process  which  will  be  used  for  validating  compliance  with  the  standard,  (4) 
define  the  procedure  which  will  be  used  to  approve  and  review  the  operations  of 
conformance  testing  sites,  and  (5)  establish  a conformance  testing  service  at  selected  sites 
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within  the  Product  Data  Exchange  Network.  The  National  PDES  Testbed  will  work  with 
other  outside  organizations  to  promote  and  accelerate  the  development  of  a conformance 
testing  service  in  order  to  support  the  objectives  of  the  DoD  Computer-aided  Acquisition 
and  Logistic  Support  (CALS)  Program. 

The  initial  focus  of  the  conformance  testing  service  will  be  to  implement  conformance 
tests  for  a single  STEP  application  protocol.  An  application  protocol  defines  the  context 
for  the  use  of  product  data  and  specifies  the  use  of  the  standard  in  that  context  to  satisfy 
an  industrial  need.  An  example  of  a context  for  the  use  of  product  data  is  process 
planning.  An  application  protocol  contains  an  abstract  test  suite  which  is  an  English 
language  definition  of  a function  or  capability  that  is  to  be  tested.  The  abstract  test  suite 
is  used  to  create  an  executable  test  suite.  The  executable  test  suites  are  computer 
instructions  written  in  an  appropriate  computer  language  which  implement  the  test 
definition  contained  in  the  abstract  test  suite.  The  conformance  testing  system  will  use 
the  executable  test  suite  to  exercise  a Client's  product  to  determine  whether  the  function 
or  capability  required  by  the  application  protocol  has  been  implemented. 

The  conformance  testing  service  will  be  used  by  implementors,  government  agencies, 
and  academia  to  ensure  that  commercially-developed  systems  conform  to  the 
requirements  of  the  appropriate  STEP  application  protocol(s).  This  project  will  work 
with  appropriate  organizations  (to  be  determined)  in  order  to  define  the  conformance 
assessment  process.  This  process  will  be  carried  out  by  an  approved  Testing  Laboratory 
for  a specific  Client. 

The  conformance  testing  service  development  effort  will  be  closely  coordinated  by  the 
National  PDES  Testbed  in  conjunction  with  the  International  Organization  for 
Standardization's  (ISO)  STEP  standards  on  conformance  testing  and  international 
activities  currently  underway  within  the  European  Community.  It  is  critical  that  the 
United  States  aggressively  position  itself  among  the  leaders  in  conformance  testing  of 
STEP  application  protocol  implementations  in  order  to  protect  U.S.  interests  in  the 
product  data  technology  arena. 
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Achieving  the  goals  of  product  data  sharing  depends  upon  conformance  testing.  It  is 
essential  that  system  implementations  can  be  tested  to  determine  whether  they  conform 
to  the  relevant  Standard  for  the  Exchange  of  Product  Model  Data  (STEP)  specifications. 
Standard  test  suites  should  be  available  for  use  by;  suppliers  or  implementors  (for  use 
in  self-testing),  users  of  product  data  exchange  products  (for  testing  vendor's  products), 
or  by  other  third-party  testing  organizations  (for  independent  testing  purposes).  The 
CAli  Test  Network  and  the  proposed  Product  Data  Exchange  Network  are  potential 
users  of  the  conformance  testing  service  established  by  this  project. 

The  National  Institute  of  Standards  and  Technology  (NIST)  has  historically  played  an 
important  role  in  the  development  of  conformance  testing  services  and  conformance  test 
suites  which  are  used  to  determine  whether  or  not  information  technology  products 
comply  with  appropriate  standards.  The  Omnibus  Trade  and  Competitiveness  Act  of 
1988  was  passed  by  Congress  and  signed  into  law  on  August  23, 1988.  Under  this  Act, 
NIST  was  given  the  charter ".  . . to  assist  industry  in  the  development  of  technology  and 
procedures  needed  to  improve  quality,  to  modernize  manufacturing  processes,  to  ensure 
product  reliability,  manufacturability,  fimctionality,  and  cost-effectiveness,  and  to 
facilitate  the  more  rapid  commercialization  ...  of  products  based  on  new  scientific 
discoveries"  [OTCA88].  Conformance  testing  helps  ensure  better  reliability,  functionality, 
and  more  rapid  commercialization  of  products. 

It  is  essential  that  implementation  validation  services  are  available  worldwide,  using 
procedures  which  are  acceptable  to  the  cooperating  countries.  Implementation  validation 
is  the  complete  process  of  accomplishing  all  conformance  testing  activities  necessary  to 
enable  the  conformance  of  an  implementation  or  a system  to  a standard  to  be  assessed 
(the  conformance  assessment  process);  when  the  fulfillment  by  an  implementation  under 
test  (lUT)  of  all  requirements  specified  is  demonstrated,  optionally  issuing  a certificate 
[ISOl-90]. 

The  goal  of  the  Conformance  Testing  Service  thread  is  to  develop  and  demonstrate  a 
conformance  testing  service  which  can  be  used  to  ensure  STEP  implementations  within 
DoD  comply  with  accepted  application  protocol  standard(s).  Conformance  testing 
methods  shall  be  applied  only  to  implementations  of  an  application  protocol.  An 
application  protocol  defines  the  context  for  the  use  of  product  data  and  specifies  the  use 
of  the  standard  in  that  context  to  satisfy  an  industrial  need  [ISOl-90]. 

To  achieve  the  above  goals,  the  project  has  the  following  objectives: 

• Promote  the  development  of  a conformance  testing  service  within  which  to 
operate  a conformance  testing  system  for  application  protocol  implementation 
testing. 
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• Establish  an  international  definition  and  acceptance  of  a conunon  testing 
methodology,  and  appropriate  test  methods  and  procedures. 

• Identify  the  tools  that  are  required  to  implement  a conformance  testing 
system. 

• Demonstrate  a successful  conformance  testing  service  for  a single  application 
protocol. 

• Minimize  the  need  for  repeated  conformance  testing  of  the  same  system 
through  wide  acceptance  of  conformance  test  reports  produced  by  different 
Testing  Laboratories. 

• Achieve  an  adequate  level  of  confidence  in  the  executable  test  suites  as  a 
guide  to  conformance. 

• Achieve  comparability  between  the  results  of  the  corresponding  tests  applied 
in  different  places  at  different  times. 

• Achieve  repeatability  of  results  for  the  same  executable  test  case  (ETC) 
applied  against  a given  implementation. 

The  development  of  a conformance  testing  service  will  require  a significant  investment 
in  management  and  technical  staff  resources,  software,  equipment,  and  facilities.  These 
costs  will  be  compounded  by  the  number  of  executable  test  suites  required  for  each 
application  protocol  which  is  needed  by  the  U.S.  Department  of  Defense. 

Once  such  a conformance  testing  service  is  established  for  STEP  implementations,  it 
must  be  made  consistent  across  national  boundaries.  In  the  past,  strong  coordination 
and  cooperation  with  other  countries  have  proven  to  be  the  most  economical  approach 
to  the  development  of  conformance  testing  systems  and  services.  This  approach  has 
facilitated  a more  rapid  development  of  conformance  testing  services  for  other 
international  standards. 
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2 Overview  of  the  Conformance  Testing  Service 

Technically  sound  national  and  international  standards  are  needed  to  preserve  open 
competition  in  international  markets  and  to  support  increased  productivity  and  delivery 
of  services  at  reduced  cost.  Through  standards,  users  are  provided  with  off-the-shelf, 
compatible  hardware,  software,  and  communications  products  for  computer  and  related 
telecommunications  systems.  Implementation  validation  of  these  computer  products 
claiming  conformance  with  standards  further  reduces  risks  and  uncertainties  to  vendors 
and  users.  Uniform  conformance  testing  procedures  should  be  employed  to  perform  this 
implementation  validation  for  a more  consistent  comparison  of  results  among  products 
[SJK88]. 

The  most  commonly  used  test  method  for  conformance  testing  computer  software 
products  is  falsification  testing.  Falsification  testing  is  a test  method  which  does  not 
guarantee  conformance  to  a standard  but  is  developed  to  find  errors  in  the 
implementation.  If  errors  are  found,  one  can  correctly  deduce  the  implementation  does 
not  conform  to  the  standard;  however,  the  absence  of  errors  does  not  necessarily  imply 
the  converse.  Thus,  conformance  to  a test  suite  alone  does  not  guarantee 
interoperability.  Conformance  testing  does  give  potential  buyers  of  a product  the 
confidence  that  a certified  implementation  has  the  required  capabilities  and  that  its 
behavior  conforms  consistently  [ISO31-90]. 

2.1  What  Needs  to  be  Done? 

Steps  in  developing  a conformance  testing  service  are  characterized  by  the  following 
major  activities: 

• Selecting  a STEP  application  protocol  and  its  supporting  abstract  test  suite. 

• Generating  an  executable  test  suite  using  the  abstract  test  suite. 

• Extending,  if  necessary  and  appropriate,  the  National  PDES  Testbed  model 
validation  hardware  and  software  system  to  provide  the  capabilities  for 
application  protocol  conformance  testing. 

• Defining  the  conformance  assessment  process  carried  out  by  a Testing 
Laboratory  for  a given  Client. 

• Providing  a conformance  testing  service  for  CALS  implementations,  other 
implementors,  government  agencies,  and  academia  for  the  chosen  application 
protocol. 
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2.2  Is  Keutrality  Important? 

A conformance  testing  service  must  be  unbiased  in  its  evaluation  of  the  conformance  of 
a product  to  a particular  standard.  Furthermore,  the  institutional  framework  in  which 
a conformance  testing  service  operates  must  be  neutral  to  ensure  that  all  Clients  are 
treated  fairly.  The  organization,  which  sets  up  the  conformance  testing  service,  must  not 
have  a vested  interest  in  the  implementation  of  any  particular  vendor. 

NIST  is  an  appropriate  institution  to  establish  the  ground  rules,  guide  the  development 
of  a conformance  testing  system,  identify  the  appropriate  Testing  Laboratories,  and  write 
the  procedures  for  a conformance  testing  service.  NIST  has  prior  experience  from 
various  other  established  conformance  testing  programs,  e.g.,  ^e  language  compiler 
validation  testing  program  (Pascal,  Fortran,  ADA),  graphics  conformance  testing 
programs  (GKS,  CGM,  PHIGS),  and  ihe  Database  Language  SQL  conformance  testing 
program.  Other  conformance  testing  programs  still  under  development  have  also 
contributed  to  NIST's  "lessons  learned,"  e.g.,  GOSIP  and  POSIX.  NIST  also  participates 
actively  in  the  national  and  international  STEP  conformance  testing  arenas.  liie  National 
PDES  Testbed  at  NIST  has  a vested  interest  in  providing  conformance  testing  services 
for  STEP  implementations  in  support  of  the  DoD  CALS  Program. 

2.3  How  Will  the  National  PDES  Testbed  Establish  a Conformance  Testing  Service? 

The  National  PDES  Testbed  will  provide  oversight  of  the  conformance  testing  service. 
It  will  establish  appropriate  conformance  testing  policies,  procedures,  and  testing 
laboratory  accreditation  requirements.  The  resources  necessary  to  develop  the 
conformance  testing  system  and  staff  the  Testing  Laboratories  which  conduct  the  actual 
conformance  testing  services  will  be  identified.  The  basic  steps  required  to  develop  the 
service  are  outlined  below  (see  section  3 for  a more  detailed  discussion): 

• Identify  the  application  protocol  implementation  priority 

• Promote  the  development  of  conformance  testing  systems 

• Identify  and  approve  candidate  Testing  Laboratories 

• Develop  conformance  testing  procedures  and  data 

• Convey  conformance  testing  systems  to  laboratory  sites 

As  new  application  protocols  are  specified  for  different  product  data  technology  areas, 
it  is  likely  that  new  laboratories  will  be  identified.  For  simplicity,  the  procedures  and 
discussion  outlined  below  address  a single  STEP  application  protocol.  The  same 
technical  processes  of  developing  executable  test  suites  or  evaluating  existing  executable 
test  suite  generators  would  be  repeated  for  each  and  every  application  protocol.  The 
conformance  testing  service  itself  and  administrative  processes  of  running  a conformance 
testing  service  woiild  remain  constant.  The  level  of  investment  required  to  establish  a 
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conformance  testing  service  for  a single  application  protocol,  as  well  as  the  resources 
needed  for  each  additional  application  protocol,  is  still  difficult  to  assess. 

2.4  How  Will  the  Service  Operate? 

The  operation  of  the  conformance  testing  service  is  based  upon  three  key  organizational 
elements.  These  elements  will  have  varying  degrees  of  responsibility  throughout  the 
development  and  the  operation  of  the  conformance  testing  service  (see  figure  1): 

Client  - The  organization  that  submits  an  implementation  for  conformance  testing 
[ISO31-90a].  This  is  normally  a company  which  develops  a product  for 
commercial  use. 

Testing  Laboratory » An  organization  that  carries  out  the  conformance  assessment 
process.  This  can  be  a third  party,  a user  organization,  and  an  administration 
[ISO31-90a]. 

Certification  Body  - An  impartial  body,  governmental  or  non-govemmental, 
possessing  the  necessary  competence  and  reliability  to  operate  a certification 
system,  and  in  which  the  interests  of  all  parties  concerned  with  the  function  of  the 
system  are  represented  [ISO31-90a]. 

These  roles  will  be  carried  out  as:  the  Client-normally  a system's  developer  who  is  a 
potential  or  active  contractor  for  DoD;  the  Testing  Laboratory—an  approved  independent 
testing  site;  and  the  Certification  Body-the  National  PDES  Testbed. 

The  role  of  the  Client  is  independent  of  any  conformance  testing  service  development. 
The  Client  has  to  progress  through  the  major  milestones  of  implementation  development 
as  depicted  in  figure  1.  During  the  early  stages  of  implementation  development,  the 
only  interest  the  Client  may  have  in  conformance  testing  is  to  obtain  and  understand  the 
standard  abstract  test  suite  (ATS).  In  later  stages  of  development,  the  Client  may  want 
to  acquire  a set  of  executable  test  data  files  from  the  Testing  Laboratory  for  "debugging" 
his  implementation  or  performing  implementation  "prevalidation";  otherwise,  the  Client 
keeps  a low  profile  in  the  scheme  of  the  conformance  testing  system  and  conformance 
testing  service  development  process.  The  Client's  role  is  as  a user  of  the  conformance 
testing  service. 

Once  a conformance  testing  system  is  set  up  and  evaluated,  the  usual  proforma  is  that 
a conformance  testing  service  will  become  self-sustaining.  Costs  associated  with  such 
a service  are  usually  carried  by  the  Testing  Laboratory  and  the  Client.  Typically,  the 
Testing  Laboratory  must  cover  the  costs  associated  with  setting  up  the  service,  getting 
approval  from  the  Certification  Body,  and  maintaining  that  approval.  The  Client  must 
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Figure  1.  Conformance  Testing 
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cx)ver  the  costs  associated  with  having  his  implementation  evaluated  and  a test  report 
generated. 

The  establishment  of  a conformance  testing  service  will  require  the  completion  of  all  of 
the  tasks  described  in  section  3 of  this  document.  Once  the  service  is  established,  the 
scenario  a Client  goes  through  is  normally  as  follows: 

1.  Request  Conformance  Testing  - The  Client  provides  information  and  supporting 
materials  to  ^e  Testing  Laboratory  in  a Letter  of  Request.  This  letter  would 
include  such  information  as:  product  identification,  including  version;  host  and 
target  configurations,  operating  systems,  program  language  and  other  appropriate 
system  information;  a Client  representative  to  contact  for  conformance  testing; 
and  the  implementation  of  a standard  application  protocol  against  which  the 
Client  is  applying  for  conformance  testing  (perform  prevalidation  on  the 
implementation) . 

2.  Prepare  for  Testing  - The  Client  prepares  the  administrative  and  more  detailed 
technical  description  of  the  implementation  which  will  undergo  testing.  The 
Testing  Laboratory  prepares  the  site  for  testing  and  selects  the  appropriate 
abstract  and  executable  test  cases.  At  this  time,  the  Client  may  also  request  a set 
of  executable  test  data  files  to  prepare  the  implementation  prior  to  actual  testing 
(perform  pre validation  on  the  implementation). 

3.  Conduct  Testing  Operations  - The  Testing  Laboratory  coordinates  a suitable  date 
for  conformance  with  the  Client.  The  Testing  Laboratory  reviews  all  details 
associated  with  the  system  under  test  (SUT).  The  actual  executable  test  suite  to 
be  used  for  the  conformance  testing  is  generated.  After  this  point  in  the  process, 
the  Client  cannot  change  the  scope  of  the  conformance  assessment.  The  tests  are 
run  and  the  results  are  recorded. 

4.  Analyze  Results  - For  each  executable  test  case,  a verdict  of  "pass,”  "fail,"  or 
"inconclusive"  is  established.  (Inconclusive  will  only  be  used  on  those  rare 
occasions  when  neither  a clear  pass  nor  fail  has  resulted.)  The  Testing  Laboratory 
analysis  is  performed  by  comparing  the  observed  test  outcomes  with  the  expected 
test  outcomes,  and  a detailed  conformance  test  report  is  generated. 

5.  Appeal  Test  Results,  If  Necessary  - If  a Client  believes  any  tests  are  in  error  or  not 
appropriate,  or  is  not  satisfied  with  the  detailed  conformance  test  report,  the 
Client  contacts  the  Testing  Laboratory  and  provides  associated  justification  for  the 
dispute.  It  is  up  to  the  Testing  Laboratory  and  Client  to  resolve  the  dispute.  (If 
the  Client  is  anticipating  applying  for  a Certificate  of  Conformance,  a copy  of  this 
letter  should  be  forwarded  to  the  Certification  Body  as  well.)  The  Testing 
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Laboratory  will  publish  the  test  report  after  the  Client  agrees  to  the  test  report 
cx)ntents. 

6.  Request  Revalidation  - When  changes  are  made  to  the  Client's  implementation, 
or  when  the  abstract  test  suite  has  been  modified,  the  Client's  implementation 
should  be  revalidated.  Revalidation  provides  a more  up-to-date  test  report  for 
marketing  the  capabilities  of  the  implementation. 

2.5  What  are  the  Issues  Involved? 

There  are  three  different  types  of  issues  from  which  the  development  of  a conformance 
testing  system  and  a conformance  testing  service  have  to  be  examined:  technical, 
adnunistrative,  and  political.  The  next  three  sections  address  these  issue  areas. 

2.5.1  Technical  Issues 

STEP  contributes  to  the  goal  of  efficient  communication  of  product  data.  It  supports 
exchange  of  data  between  computer  software  systems  with  similar  or  related  functions 
but  dissimilar  internal  data  formats  and  structures.  It  provides  neutral  information 
reference  models  and  standardized  implementation  practices  for  advanced  computer 
software  systems  to  enable  the  use  of  STEP  as  their  internal  data  format(s)  in  the  longer 
term.  In  the  "data  exchange"  role,  STEP  will  be  implemented  in  computer  system 
translators  and  referenced  in  data  delivery  agreements.  In  the  second  role,  STEP  will  be 
implemented  in  the  core  of  computer  systems  and  referenced  in  system  architecture 
documents  for  coordination  between  system  components  [ISOl-90]. 

The  technical  issues  are  associated  with  the  development  of  the  conformance  testing 
system  itself  and  the  supporting  executable  test  suite  generator.  Initial  conformance 
testing  for  STEP  is  targeted  at  file  exchange  between  two  systems.  Since  so  little  is 
currently  known  on  how  conformance  testing  will  occur  against  database 
implementations,  this  document  will  emphasize  conformance  testing  of  implementations 
used  for  file  exchange.  As  international  consensus  is  reached  on  how  conformance 
testing  will  be  approached  for  databases,  this  document  will  be  updated  to  reflect  those 
decisions.  As  commercial  implementations  are  developed  for  both  file  exchange  and 
database,  the  National  PDES  Testbed  will  be  faced  with  the  issues  associated  with 
development  and  simultaneous  support  of  multiple  conformance  testing  services. 

Conformance  Testing  Process  for  File  Exchange 

A conformance  testing  service  for  file  exchange  conformance  testing  can  be  handled  very 
similarly  to  those  of  other  standards,  e.g..  Computer  Graphics  Metafile  (CGM),  [FIP128] 
where  the  executable  test  suite  is  processed  against  an  output  data  file  from  the 
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preprocessors  or  postprocessors  (generators  and  interpreters  in  CGM).  A preprocessor 
is  a software  unit  that  translates  product  information  from  the  internal  database  format 
of  a certain  CAx  system  to  an  independent  public  domain  product  data  format  [ISOl-90]. 
(CAx  is  used  to  generically  describe  any  computer-aided  system.)  Preprocessor  analysis 
is  clearer  than  the  postprocessor  analysis,  since  in  this  case  the  results  are  a known  form 
on  which  to  work:  a STEP  data  file.  The  data  files  can  be  checked  for  syntactical 
conformance,  through  a parser,  then  compared  against  the  expected  model  as  specified 
by  the  abstract  test  suite,  as  well  as  for  semantical  conformance  as  prescribed  by  the 
application  protocol. 

A postprocessor  is  a software  unit  that  translates  product  information  from  an 
independent  public  domain  product  data  format  to  the  internal  database  format  of  a 
certain  CAx  system  [ISOl-90].  There  are  two  options  to  perform  analysis  on  the  results 
of  postprocessor  tests: 

• Output  a STEP  file  using  the  preprocessor  and  use  that  as  the  basis  for 
analysis. 

• Process  the  postprocessor  against  an  executable  test  suite  derived  from  the 
standard  abstract  test  suite. 

The  first  requires  the  preprocessor  to  have  been  tested  and  to  have  relevant  support  in 
the  context  of  each  postprocessor  test.  This  approach  does  not  address  the  problem  of 
symmetric  errors  in  the  two  processors  or  take  into  account  the  situation  where  there 
may  be  only  a postprocessor  undergoing  testing  [JO90]. 

The  second  analysis  method  should  be  adopted  whenever  possible.  Both  syntactic  and 
semantic  (structure  and  use  in  terms  of  the  properly  defined  standard  meaning) 
evaluation  can  be  p>erformed  based  on  the  requirements  of  STEP  and  the  specific 
application  protocol.  The  analysis  can  be  highly  automated  and,  therefore,  objective. 
See  figure  2 for  a simple  architecture  of  the  file  exchange  conformance  testing  system 
and  conformance  testing  system  capabilities. 

Conformance  Testing  of  a Database  Implementation 

A major  goal  of  the  STEP  effort  is  product  data  sharing  through  shared  databases. 
Components  of  a STEP  data  sharing  environment  and  the  issues  arising  from 
implementation  of  such  an  environment  are  discussed  in  "A  High-level  Architecture  for 
Implementing  a PDES/STEP  Data  Sharing  Environment"  [PI91].  The  key  to  realize  a 
heterogeneous  STEP  data  sharing  environment  is  establishing  a standard  software 
interface  between  STEP  data  storage  management  software  systems  and  applications 
producing /consuming  data  (according  to  specific  application  protocols).  Specifying  such 
an  interface  is  currently  underway  in  the  ISO  STEP  Working  Group  on  Iinplementation 
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Specifications  [DP91,  JF90].  The  interface  is  commonly  known  as  the  STEP  Data  Access 
Interface  (SDAI).  \^en  SDAI  becomes  part  of  STEP,  vendors  desiring  to  provide 
systems  which  share  data  through  databases  will  implement  the  interface  as  part  of  their 
applications  or  databases.  Applications  producing /consuming  STEP  data  will  make 
SDAI  requests  for  STEP  data  to  shared  databases  and  the  databases  will  respond  to  those 
requests  accordingly.  Thus  there  will  be  a need  for  conformance  testing  of  an 
application's  support  of  SDAI  in  the  context  of  the  application  protocol  with  which  it 
intends  to  comply.  Additionally,  there  will  be  a need  for  conformance  testing  of  a 
database's  ability  to  provide  STEP  data  according  to  requests  of  SDAI  by  applications. 

Since  SDAI  is  a software  interface,  it  will  in  many  respects  simplify  the  task  of  executing 
conformance  tests  against  both  applications  and  databases.  CT  system  processors  can 
be  written  against  the  SDAI  specification  rather  than  against  proprietary  application 
interfaces  or  database  vendor's  proprietary  storage  design  for  STEP  data. 

2.5.2  Administrative  Issues 

Procedures  are  needed  which  encourage  cooperation  by  national  and  international 
certification  and  standardization  bodies  to  operate  under  a common  strategy.  These 
institutions  should  mutually  recognize  conformance  testing  results  and  certificates  for 
STEP  application  protocol  implementations.  The  scope  of  current  cooperative  activities 
in  other  conformance  testing  services  ranges  from  the  bilateral  recognition  of  the  Testing 
Laboratories  (e.g.,  a Memorandum  of  Understanding)  to  the  use  of  internationally 
standardized  testing  procedures. 

The  conformance  testing  service  under  which  the  National  PDES  Testbed  plans  to 
operate  will  be  coordinated  with  both  the  International  Organization  for  Standardization 
(ISO)  STEP  conformance  testing  procedural  standards  and  international  activities 
currently  underway  within  the  European  Conununity.  Such  a multi-national 
conformance  testing  service  agreement  already  exists  for  CAD/CAM  exchange  standards 
between  several  European  countries  [CTS88]. 

Figure  3 provides  an  illustration  of  the  relationships  between  the  various  authorities 
involved  in  a conformance  testing  service.  The  Client  and  Testing  Laboratory  have  been 
previously  introduced;  their  role  within  a conformance  testing  service  will  be  briefly 
outlined  below.  Beyond  the  Client  and  the  Testing  Laboratory,  additional  authorities 
associated  with  a conformance  testing  service  are  the  Certification  Body  and  (optionally) 
the  Accreditation  Body. 

The  members  of  the  Certification  Body  should  be  organizationally  separate  from  the 
Testing  Laboratory  staff.  Certification  Body  responsibilities  include  approval  of  the  test 
method  and  executable  test  suite  generator,  approval  of  the  conformance  testing 
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procedures,  and  definition  of  the  criteria  for  recognizing  a Testing  Laboratory.  Figure 
4 shows  the  various  methods  for  Testing  Laboratory  selection  and  recognition. 

If  economics,  conformance  testing  requirements,  and  politics  warrant  accreditation  of 
Testing  Laboratories,  an  Accreditation  Bod/s  services  would  be  employed.  An 
Accreditation  Body  conducts  and  administers  a laboratory  accreditation  scheme  and 
grants  accreditation.  This  assessment  of  a Testing  Laboratory  may  be  carried  out  in  full 
or  in  part  by  the  Certification  Body  instead.  Testing  Laboratory  accreditation  is  the 
formalized  initial  and  continuing  process  of  ensuring  a Testing  Laboratory  is  competent 
to  carry  out  specific  types  of  tests.  The  term  "laboratory  accreditation"  covers  the 
recognition  of  both  the  technical  competence  and  the  impartiality  of  a Testing  Laboratory 
[ISO31-90].  Accreditation  is  normally  awarded  following  successful  laboratory 
assessment,  the  steps  as  shown  in  figure  4 [SJK88],  and  is  followed  up  by  appropriate 
surveillance.  It  is  likely  that  some  of  the  Testing  Laboratories  associated  with  the 
Product  Data  Exchange  Network  [SF90]  will  be  interested  in  seeking  acceptance  as 
Testing  Laboratories  under  the  STEP  conformance  testing  service. 
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2.5.3  Political  Issues 

The  Commission  of  the  European  Communities  (CEC)  is  acting  swiftly  and  deliberately 
to  turn  twelve  European  countries’  into  a single,  integrated  market  of  320  million  people 
by  the  end  of  1992.  The  basis  for  this  effort  is  a 1985  European  Community  (EC)  White 
Paper  entitled  "Completing  the  Internal  Market."  That  paper  sets  a timetable  for  the 
measures  needed  to  ensure  the  free  circulation  of  persons,  products,  services,  and  capital 
among  the  twelve  member  states.  The  EC  already  initiated  a program  to  eliminate  the 
many  differing  national  standards  and  technical  regulations;  it  has  drawn  up  more  than 
200  EC  directives  aimed  at  harmonizing  the  various  national  requirements. 

The  task  of  establishing  European  technical  standards  for  products  will  be  left  to 
European  standardization  bodies  set  up  by  industry^  and  other  European  governments. 
In  the  absence  of  standards  to  be  harmonized  or  dted,  the  CEN-type  (European 
Committee  for  Standardization)  organizations  are  to  adopt  standards  based  on 
international  standards  developed  by  such  groups  as  the  International  Organization  for 
Standardization  (ISO)  and  the  International  Electrotechnical  Commission  (lEC)  [PC88]. 

In  support  of  the  EC  economic  goal  of  1992,  the  United  Kingdom,  Germany,  and  France 
are  collaborating  on  a common  objective  for  CAD /CAM  systems  data  exchange 
interfaces:  to  provide  harmonized  testing  services  within  Europe  for  the  transfer  of 
product  models  between  different  CAD  systems  using  neutral  formats.  Started  in 
December  1988,  this  service  will  be  ready  for  use  in  time  for  the  approval  of  ISO  STEP 
and  offer  interim  services  for  the  U.S.-based  Initial  Graphics  Exchange  Specification 
(IGES),  the  German-based  (VDAFS),  and  the  French-based  (SET).  Both  the  development 
of  the  test  tools  and  the  establishment  of  operational  Testing  Laboratories  offering  testing 
services  are  covered  in  the  project.  Established  as  a multi-phased  project,  the  current 
phase  is  part  of  a broader  program  Conformance  Testing  Services  Two,  commonly 
known  as  CTS2.  In  addition  to  the  four  full  nation  partners,  there  is  one  associate 
partner,  Sweden.  Beyond  national  commitment  of  resources  and  money,  the  EC 
members  are  funded  by  the  Commission  of  the  European  Community  [CTS88][FWK90]. 

National  PDES  Testbed  staff  believe  that  it  is  critical  to  actively  participate  in  this 
cooperative  international  effort.  As  such,  a Testbed  representative  provided  a briefing 


’The  Member  States  of  the  EC  include  Belgium,  Denmark,  France,  Germany  (F.R.),  Greece,  Ireland,  Italy, 
Luxembourg,  the  Netherlands,  Portugal,  Spain,  and  the  United  Kingdom. 

^Such  as  the  European  Committee  for  Standardization  (CEN)  which  promotes  European  regional  standardization 
in  the  non-electrotechnical  field;  European  Committee  for  Electrotechnical  Standardization  (CENELEC):  and  European 
Conference  of  Postal  and  Telecommunications  Administrations  (CEPT)  which  publishes  recommendations  to 
harmonize  and  improve  administration  and  operational  services. 
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paper  at  the  CAD/ CAM  CTS2  meeting  in  September  1990  as  part  of  a first  attempt  to 
collaborate  in  the  conformance  testing  service  activities.  Initially  focussing  on  ICES, 
NIST  has  extended  its  activities  through  the  National  PDES  Testbed  into  the  STEP 
conformance  testing  service  program  and  policy  development.  Having  a nationally- 
based  conformance  testing  service  helps  U.S.  industry  remain  competitive  at  home  and 
abroad.  Through  NIST  collaboration  with  the  EC  1992  conformance  testing  activities  and 
the  establishment  of  an  internationally  cooperative  conformance  testing  service,  CALS- 
supported  implementations  are  afforded  better  opportunity  for  accepted  use  and 
distribution  worldwide. 

Resolution  of  the  above  technical,  administrative,  and  political  issues  is  critical  to  a 
successful  STEP  conformance  testing  service.  As  we  progress  through  the 
standardization  and  implementation  of  STEP  application  protocols,  we  may  identify 
more  issues,  as  well  as  realize  solutions  to  those  issues  stated  above. 
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The  following  tasks  need  to  be  accomplished  over  the  time  period  specified  in  this  plan. 
In  order  to  accomplish  such  a rigorous  agenda,  the  National  PDES  Testbed  will  be 
dependent  on  development  activities  carried  out  by  private  enterprise,  industry 
associations,  and  the  academic  community.  Other  than  those  efforts  associated  with 
CTSl  below,  the  tasks  of  this  plan  apply  to  a single  application  protocol. 

CTS  0 Prepare  Technical  Development  Plan 

This  document  is  the  CTS  0 deliverable. 

CTS  1 Identify  DoD  Application  Protocol  Implementation  Priority 

Determine  what  the  U.S.  DoD  priorities  are  for  the  various  standardized  application 
protocols.  Those  application  protocols  from  the  STEP  specification  200  series  parts  will 
be  exanrdned  and  should  be  considered  only  if  they  are  at  the  draft  international 
standard  level.  Commercial  product  developers  are  not  likely  to  begin  intense 
development  activities  until  this  level  of  specification  is  reached,  i.e.,  when  the  standard 
is  considered  more  technically  stable.  It  also  is  more  prudent  for  the  conformance 
testing  system  tool  development  efforts  to  begin  only  during  the  later  stages  of  the 
committee  draft  cycle  of  any  given  application  protocol.  Steps  appropriate  for  gathering 
the  necessary  information  indude:  hold  workshops,  evaluate  AP  proposals,  and  select 
the  first  application  protocol  for  conformance  testing. 

CTS  2 Develop  Conformance  Testing  Procedures 

Based  on  the  STEP  parts  which  address  conformance  testing,  develop  layout  and  content 
requirements  of  test  logs,  summary  and  detailed  conformance  test  report  formats, 
document  all  responsibilities  associated  with  a Testing  Laboratory,  and  establish 
appropriate  fee  structure  to  make  a conformance  testing  program  self-sufficient. 

CTS  3 Develop  Conformance  Testing  System 

This  task  and  its  associated  sub-tasks  will  identify  requirements  for  a conformance 
testing  system  in  support  of  an  effective  conformance  testing  service,  and 
acquire/ develop  the  appropriate  hardware  and  software  to  develop  the  system. 

CTS  3.1  Requirements  Analysis 

Identify  requirements  and  constraints  for  the  conformance  testing  of  STEP  application 
protocols.  Given  any  implications  identified,  revisit  and  amend  where  necessary,  the 
schedule  and  milestones  for  developing  a conformance  testing  system  and  conformance 
testing  procedures. 
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CTS  3^  Prepare  Conformance  Testing  System  Design 

Develop  a conformance  testing  system  design  plan  which  specifies  the  actual  hardware 
and  software  configuration  required  to  provide  useful  analysis  for  a conformance  testing 
service.  Such  aspects  as  utility  requirements,  base  band  networking,  and  integration  will 
be  considered.  Identify  existing  test  tools,  executable  test  suite  generators,  and 
documentation. 

CTS  3.3  Acquire  Supporting  Hardware,  Software,  and  Equipment 

Based  on  the  design  plan,  obtain  (if  necessary)  any  additional  commercial  software  and 
hardware  through  donation,  loan,  or  acquisition.  Note:  The  effort  required  for  this  task 
may  be  minimized  if  the  National  PDES  Testbed  STEP  Validation  Testing  System 
configuration  can  be  either  used  "as  is",  reconfigured,  or  used  as  a basis  to  provide  the 
major  components  for  the  conformance  testing  system. 

CTS  3.4  Acquire  an  Executable  Test  Suite  Generator  for  the  Application  Protocol 

It  is  necessary  to  be  able  to  generate  an  executable  test  suite  based  on  the  abstract  test 
suite  that  supports  the  application  protocol.  The  deliverable  for  this  task  will  be  an 
executable  test  suite  generator  acquired  in  one  of  two  ways:  by  accepting  an  already 
existing  executable  test  suite  generator  developed  by  another  organization  or  country; 
or  by  having  the  executable  test  suite  generator  developed.  If  the  latter  is  necessary,  it 
is  much  more  labor  intensive  and  expensive;  however,  the  options  available  will  be 
determined  by  the  U.S.  DoD  user  requirements  and  prioritization  of  application 
protocols. 

CTS  3.4.1  Accept  Existing  Executable  Test  Suite  Generator  and  Support  Tools 

Assuming  that  there  is  already  an  existing  executable  test  suite  generator,  support  tools, 
and  documentation,  the  National  PDES  Testbed  will  obtain  access  to  them  via  contract, 
loan,  or  purchase.  The  National  PDES  Testbed  will  evaluate  the  quality  and 
completeness  of  the  products  and  arrange  licensing  agreements  for  those  generators 
whi^  are  proprietary.  These  activities  would  be  closely  monitored  and  coordinated 
through  the  other  international  efforts  to  ensure  that  U.S.  decisions  would  be  supported 
worldwide.  The  conclusion  of  this  task  will  be  that  the  accepted  generator  and  all 
support  tools  are  up  and  running  on  the  conformance  testing  system.  Any  necessary 
debugging  after  beta  testing  will  be  completed.  At  a minimum,  the  license  agreement 
shall  allow  the  National  PDES  Testbed  to  obtain  the  generator  and  make  it  available  to 
any  interested  party.  There  shall  be  no  national  restrictions  on  its  use,  and  an 
appropriate  fee  structure  for  reimbursement  to  the  generator  owner  shall  be  developed. 
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CIS  3.4.2  Develop  an  Executable  Test  Suite  Generator  and  Support  Tools 

Deliverables  associated  with  the  development  of  the  executable  test  suite  generator  based 
on  the  appropriate  STEP  abstract  test  suite  include:  an  automated  executable  test  suite 
generator.  Express  parser,  written  specifications  for  source  code  development,  developed 
source  code,  and  supporting  documentation.  This  task  will  be  completed  only  after  the 
executable  test  suite  generator  has  been  completely  tested  and  released  for  use. 

CTS  3.5  Distribute  Test  Suite  Generator 

The  National  PDES  Testbed  shall  announce  the  executable  test  suite  generator's 
availability.  Included  in  the  announcement  shall  be:  NIST  point  of  contact,  how  to 
obtain  the  test  suite  generator  and  support  tools,  and  how  to  obtain  an  information 
package  for  more  details.  Deliverables  from  this  task  will  include:  an  information 
package  which  describes  the  generator  and  conformance  testing  service,  fees,  and  source 
of  avc^ability.  Anticipated  users  of  the  generator  would  include  commercial  vendors, 
industry,  government,  and  academia. 

CTS  4 Identify  Testing  Laboratory  Candidates 

Notify  institutions  which  might  have  an  interest  in  becoming  conformance  Testing 
Laboratories.  Develop  an  information  package  for  candidate  laboratories  which  outlines 
costs  and  benefits.  As  an  option,  accredit  candidate  laboratories. 

CTS  4.1  Determine  Appropriate  Method  of  Laboratory  Selection 

Develop  a report  which  outlines  the  technical,  managerial  and  administrative 
requirements  necessary  for  becoming  a Testing  Laboratory. 

CTS  4.2  Accredit  Testing  Laboratory  (Optional) 

Develop  criteria  to  evaluate  Testing  Laboratory  qualifications.  Advertise  for  applicants 
and  perform  assessment(s).  An  evaluation  report  is  written  at  the  end  of  each  review, 
and  a presentation  is  given  to  the  laboratory's  management.  When  a laboratory  is 
approved  and  it  accepts  the  task,  the  National  PDES  Testbed  will  publish  an 
announcement  stating  the  availability  of  the  accredited  Testing  Laboratory. 

CTS  5 Convey  Conformance  Testing  System  to  Multiple  Sites 

Once  the  conformance  testing  system  has  demonstrated  prototype  conformance  testing 
of  the  selected  application  protocol,  qualified  Testing  Laboratory(ies)  have  been 
identified,  and  the  policies  and  procedures  for  a conformance  testing  service  are 
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established,  the  National  PDES  Testbed  will  assist  in  the  conveyance  of  the  conformance 
testing  system  technology  to  the  identified  Testing  Laboratory  sites.  This  task  will 
include  ^e  sharing  of  all  software  specification  manuals,  government-owned  testing 
tools,  procedure  manuals;  and  arranging  site  licensing  for  those  products  used  within 
the  conformance  testing  service  which  are  proprietary. 

CTS  6 Provide  an  Implementation  Validation  Service 

Implementation  validation  is  the  complete  process  of  accomplishing  all  conformance 
testing  activities  necessary  to  enable  the  conformance  of  an  lUT  to  a standard  to  be 
assessed.  Such  a service  may  end  either  at  the  issuing  of  a conformance  testing  report, 
or  if  deemed  appropriate  and  the  lUT  has  met  all  the  requirements,  issuing  a certificate. 
Such  a validation  service  will  be  based  on  STEP  ISO  CD  10303-31,  "Conformance  Testing 
Methodology  and  Framework:  General  Concepts."[ISO31-90a]  This  task  will  include 
writing  the  policy  and  procedures  for  a conformance  testing  service;  developing  an 
appropriate  fee  structure  for  the  service;  advertising  availability,  and  beginning  a service. 

Figure  5 provides  a work  breakdown  structure  for  the  above  tasks.  Figure  6 is  a Gantt 
chart  which  graphically  depicts  the  schedule  for  this  plan.  Figure  7 is  a table  of 
deliverables  associated  with  the  tasks  which  have  been  identified. 
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Figure  5.  Work  Breakdown  Structure  for  Conformance  Testing  Service  Project 
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Figure  6.  Schedule  for  the  Conformance  Testing  Service  Project 
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4 Resources 


4.1  Personnel 

Project  Manager.  Responsible  for  developing  work  plans,  defining  tasks,  and  monitoring 
milestones  and  activities.  Identifies  appropriate  resources  for  completing  each  task. 
Coordinates  activities  between  the  Testing  Laboratory,  Client  and  Certification  Body.  In 
the  absence  of  a Certification  Body,  ensuring  publication  of  resultant  conformance  test 
reports.  Serves  as  test  site  coordinator  on  a continual  basis. 

Lead  Analyst.  Responsible  for  analyzing  functional  requirements  of  the  abstract  test 
suite.  Translates  these  functional  requirements  into  system  design  specifications  and  an 
executable  test  suite  generator  for  the  conformance  testing  system. 

Software  Engineer  and  Programmer(s).  Responsible  for  developing  software  utilities  in 
the  appropriate  programming  languages  and  database  languages  for  the  use  of 
conformance  testing  implementations  of  STEP  application  protocols. 

Lawyer.  Review  and  approve  all  written  contractual  arrangements,  memoranda  of 
understanding,  and  publication  arrangements  for  conformance  test  reports.  Review  and 
approve  the  certificate  format. 

Testing  Site  Coordinator.  Responsible  for  the  administrative  and  budgetary  details  of 
nmning  a conformance  testing  service  Testing  Laboratory.  Publishes  announcements  for 
availability  of  the  application  protocol  executable  test  suite  generator,  the  conformance 
testing  service,  and  the  validated  products  list  (which  lists  those  implementations  which 
have  undergone  conformance  testing). 

Software  Evaluator.  Responsible  for  evaluating  and  assessing  the  completeness  and 
accuracy  of  existing  executable  test  suite  generator  and  support  tools  for  their  application 
in  this  conformance  testing  service. 

Certification  Body  (optional).  Responsible  for  establishing  the  certification  aspects  of  a 
conformance  testing  service.  Manages  and  coordinates  the  activities  of  the  conformance 
testing  service.  Approves  the  test  method  and  associated  executable  test  suite  generator 
for  the  application  protocol.  Identifies  the  Testing  Laboratory(ies)  capable  of  performing 
a conformance  testing  service.  Establishes  the  conformance  testing  service  procedures, 
and  encourages  collaboration  with  other  foreign  national  conformance  testing  services. 
Has  signature  authority  on  any  certificates. 
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4.2  Equipment 

The  conformance  testing  service  project  will  require  access  to  computer  hardware  and 
software  which  will  support  the  operation  and  evaluation  of  conformance  testing 
systems. 

Computer  Hardware 

• 3-4  workstations 

• personal  computers 

• local,  wide  area,  and  inter-site  networks 

• laser  printers 

Software 

• file  and  data  editors 

• report  generators 

• commercial  database  management  tools 

• parsers 

• executable  test  data  files 

• executable  test  suite  generator 

• commercial  graphics  display  packages 

Given  the  leading  activities  and  required  acquisition  of  hardware  and  software  for  the 
Validation  Testing  System  [MM90],  no  additional  equipment  requirements  to  those  listed 
above  are  anticipated  for  the  Conformance  Testing  System. 

4.3  Facilities 

National  PDES  Testbed  Facility 

The  National  PDES  Testbed  will  serve  as  the  headquarters  facility  for  the 
conformance  testing  service. 

Product  Data  Exchange  Network  Facilities 

The  Product  Data  Exchange  Network  facilities  will  participate  in  a variety  of  ways 
during  the  development  and  implementation  of  a conformance  testing  service. 
Some  of  this  participation  may  include:  executable  test  suite  generator 
development  or  evaluation,  conformance  testing  system  development,  hosting 
workshops,  preparing  educational  seminars. 
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Testing  Laboratory(ies) 

Approved  Testing  Laboratories  will  carry  out  the  actual  conformance  testing  and 
generate  test  reports  against  a Client's  product. 

NIST  Conference  Room  Facilities 

Conferences,  workshops,  and  training  seminars  will  require  access  to  NIST 
conference  room  facilities. 
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7 Glossary 


The  following  terms,  definitions,  and  acronyms  are  taken  from  working  drafts  of  the  ISO 
TC184/SC4  community  [ISOl-90],  [ISO31-90a]. 

Abstract  Test  Case  (ATC) 

One  or  more  files,  encapsulating  the  test  purpose,  which  provide  the  basis  from 
which  parameterized  executable  test  cases  are  derived. 


AP 

Application  Protocol 

CAx 

, Any  product-related  computer-aided  software  system 
CGM 

Computer  Graphics  Metafile 

CTS 

Conformance  Testing  Service 
Conformance  Assessment  Process 

The  complete  process  of  accomplishing  all  conformance  testing  activities  necessary 
to  determine  the  conformance  of  an  implementation  to  an  application  protocol. 

Conforming  Implementation 

An  implementation  under  test  which  satisfies  the  conformance  requirements, 
consistent  with  the  capabilities  stated  in  the  protocol  implementation  conformance 
statement. 

Conformity 

The  fulfillment  by  an  implementation  under  test  of  all  requirements  specified. 
Also  known  as  conformance. 

Executable  Test  Case  (ETC) 

A realization  of  an  abstract  test  case. 


GKS 

Graphic  Kernel  System 
GOSIP 

Government  Open  Systems  Interconnection  Profile 
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Implementation  Under  Test  (TUT) 

That  part  of  a product  which  is  to  be  studied  under  testing,  which  should  be  an 
implementation  of  one  or  more  characteristics  of  the  standard. 


PDFS 

Product  Data  Exchange  using  STEP 


PHIGS 

Programmer's  Hierarchical  Interactive  Graphics  System 
POSIX 

Portable  Operating  System  Interface  for  Computer  Environments 


Semantics 

The  relationships  of  characters  or  groups  of  characters  to  their  meanings. 


STEP 

STandard  for  the  Exchange  of  Product  model  data 
Syntax 

The  relationship  among  characters  or  groups  of  characters,  independent  of  their 
meanings  or  the  manner  of  their  interpretation  and  use. 

System  Under  Test  (SUT) 

The  computer  hardware,  software  and  communication  network  required  to 
support  the  implementation  imder  test. 

[Abstract  (ATS)  or  Executable  (ETS)]  Test  Suite 

A complete  set  of  test  cases,  possibly  combined  into  nested  test  groups,  that  is 
necessary  to  perform  conformance  testing  for  a standard  or  group  of  standards. 
Note:  This  term  is  deprecated  unless  prefixed  by  abstract  or  executable. 
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